This application is based upon and claims the benefit of the priority from the prior Japanese Application No. 2000-377839, filed Dec. 12, 2000, the entire contents of which are incorporated herein by reference.
1. Field of the Invention
The present invention relates to an image processing device for decoding digitally coded image data, and a television receiver and an image (picture) reproducing device which use the image processing device. More specifically, the present invention relates to a device for decoding image data, such as MPEG-based data, to which the decode time stamps are appended.
2. Description of the Related Art
A technique which uses the decode time stamp in decoding coded data, such as MPEG data, is disclosed in, for example, Jpn. Pat. Appln. KOKAI Publication No. 8-116532. This technique is intended to facilitate the decode operation control by writing the decode time stamp (hereinafter referred to as DTS) into a buffer together with coded image data and utilizing the DTS read from the buffer in starting frame-by-frame decoding.
That is, before starting decoding, the DTS appended to the beginning of a frame is compared with the count in a system time clock (STC) counter in the decoding device. When a match occurs, the corresponding image data is read from the buffer and then decoded.
However, errors may occur in coded data input to the decoding device when they are received through broadcasting or played back from a recording medium. In such case, the decoding device may fail to perform the decoding operation properly or the start of decoding may be delayed.
Once the start of decoding is delayed, the restoration to the proper decoding operation becomes difficult because the DTS and the counter in the STC counter do not match. The delay of the start of decoding may cause the buffer to overflow.
In Jpn. Pat. Appln. KOKAI Publication No. 11-225332 is disclosed a technique which involves attaching identification information (hereinafter referred to as ID) to each frame of image data, creating a table in which IDs are mapped to DTSs, image data storage locations in the buffer, and coding types, and managing the image data decoding operation using that table. According to this technique, the STC is produced based on the program clock reference value (hereinafter referred to as PCR) and the resulting STC is compared with the DTS to determine the ID of a frame to be decoded. A frame of image data the ID of which corresponds to the DTS that matches the STC is read from the buffer and then decoded.
The image decoding method and the decoding device disclosed in the above Patent Publication will described below with reference to FIGS. 1, 2, and 3.
FIG. 1 is a block diagram of the image data decoding device for decoding coded image data (for example, MPEG-2 transport stream) using PCR and DTS.
A coded bit stream with temporal information, such as an MPEG-2 transport stream (TS), is input to a demultiplexer (DMUX)/PCR extractor (hereinafter abbreviated as the PCR extractor) 11, where the PCR value required to produce the reference time (STC) is extracted from the bit stream. The extracted PCR is sent to an STC counter 12, where a comparison is made between the STC (count) and the PCR. When there is a difference between them, the STC is reset to the PCR; namely, the STC is revised to the reference time on the encoder side.
Also, in the PCR extractor 11 a coded bit stream with DTSs, such as an MPEG-2 packetized elementary stream (hereinafter referred to as PES), is separated from the input data and then sent to a DTS extractor 13. The DTS is extracted in the DTS extractor 13 and then applied to an ID/DTS/storage location holding circuit 14.
The data input to the DTS extractor 13 is sent to an ID adder 15, where the header information of each frame of image data is detected and identification information (ID) is produced for each frame on the basis of its header information. A picture type (coding type for the corresponding frame) is sent from the ID adder 15 to the ID/DTS/storage location holding circuit 14 together with the corresponding ID.
The data input to the ID adder 15 are written into an input buffer 16 in sequence. The storage location information (an address in the buffer) for each data written into the input buffer 16 is presented from the input buffer 16 to the ID/DTS/storage location holding circuit 14.
Here, the data input to the ID adder 15 may be a coded bit stream with temporal information, such as PES. However, since the temporal information needs to be removed in a decoder to be described, it is desirable that the input data be coded image data without temporal information (i.e., elementary stream: ES).
In the ID/DTS/storage location holding circuit 14 is created and stored a table in which the DTS extracted from each frame of input data by the DTS extractor 13, the storage location in the buffer 16 in which the corresponding input data has been stored, and the ID produced for the corresponding input data in the ID adder 15 are mapped (ID/DTS/storage location mapping table).
In a comparator 17, the STC from the STC counter 12 is compared with DTSs in the table in the ID/DTS/storage location holding circuit 14 to determine the presence or absence of a frame for which decoding is to begin. In the presence of such a frame, its ID is sent to a decoding controller 18.
The storage location information corresponding to the ID from the comparator 17 is read from the ID/DTS/storage location holding circuit 14 into the decoding controller 18, which in turn provides the storage location and decoding area of data to be decoded and an instruction to start decoding to the decoder 19. The decoder can decode any rectangular image area.
The decoding controller 18 makes reference to the state of the decoder 19 to determine a reproduction mode, such as normal, skip, or repeat. The normal reproduction mode is one in which each frame of image data is reproduced in succession. The skip reproduction mode is one in which frames are reproduced with some of them skipped. The repeat reproduction mode is one in which the same frame is reproduced repeatedly. In the decoder 19, data read from the specified storage location in the buffer is decoded.
In MPEG-2, there are three image compression schemes. The first scheme utilizes spatial correlation for coding (compression). In a natural image, since the values of neighboring pixels are close to each other (interpixel correlation is high), the spatial correlation in the image is utilized for coding (compression). The second scheme utilizes interframe (temporal) correlation in an image sequence. In this scheme, compression is performed by storing image information from the previous frame and representing the current frame by the difference from the previous frame. The third scheme utilizes nonuniform probabilities of occurrence of symbols in coding (compression) by the first and second schemes.
Of image data decoded by the decoder 19, frame data coded utilizing spatial correlation are written into a decode memory 20 as they are. As for frame data coded utilizing temporal correlation, data in the process of being decoded is processed using previously decoded image data from the decode memory 20 and the resultant decoded data is written into the decode memory. After decoded data have been written into the decode memory 20, the decoder 19 notifies the decoding controller 18 of the termination of decoding and the location at which readout was terminated.
To decode multi-channel image data, the same table as described previously is created for each channel in the ID/DTS/storage location holding circuit 14. In this case, using the readout termination location information from the decoder 19 the decoding controller 18 allows the decoder to decode image data on different channels on a time-sharing basis. Decoded image data for each frame is sent to a display processor.
The operation of the image data decoding device shown in FIG. 1 will be explained using a table of FIG. 2 created by the ID/DTS/storage location holding circuit 14. Each frame of image data input to the DTS extractor 13 contains a DTS. The ID adder 15 detects the frame header, then identifies the picture type (hereinafter abbreviated as PT) and creates a frame ID.
MPEG-1 and MPEG-2 provide bidirectional prediction using two frame memories; forward prediction from a past reconstructed image and backward prediction from a future reconstructed image. To implement such bidirectional prediction, MPEG-2 defines three picture types: I-picture (intra-picture), P-picture (predictive picture), and B-picture (bidirectionally predictive picture).
In FIG. 2, frame IDs are taken as Nxe2x88x921, N, and N+1 and PTs as I, P, and B. Image data have been written closely into the input buffer 16 in the order of frame Nxe2x88x921, frame N, and frame N+1. Each shaded portion in the image data indicates the header of the corresponding frame. In the ID/DTS/storage location holding circuit 14, a table is created and stored in which the ID, PT, DTS and header storage location are mapped for each frame.
In the comparator 17, the DTSs in the table are compared with STC to identify the ID of frame data to be decoded. Upon receipt of that ID, the decoding controller 18 identifies the storage location of that frame in the input buffer 16 and then instructs the decoder 19 to start decoding of the frame data in that storage location. The time at which the decoding controller 18 instructs the decoder to start decoding may be synchronous with a decode sync signal of a fixed period applied to the controller. Alternatively, the controller 18 may instruct the decoder 19 as soon as it receives ID information from the comparator 17, in which case the comparator 17 is configured to make comparison with DTSs and STC in synchronism with that decode sync signal.
An example of a decode operation will be explained by reference to FIG. 3. In addition, why the above table is needed will be described.
FIG. 3 illustrates (1) a decode operation without the table (Jpn. Pat. Appln. KOKAI Publication No. 8-116532) and (2) a decode operation with the table (Jpn. Pat. Appln. KOKAI Publication No. 11-225332).
In FIG. 3, the horizontal axis represents time. The time axis is equally divided into sections and one section is assumed here to be one-thirtieth seconds. Each interval indicated by a line with arrows is one when frame data is being decoded.
FIG. 3 shows a situation in which the decoding dO of the 0-th frame is terminated as scheduled, but the termination of decoding dl of the first frame is delayed for some cause.
In the decoding operation (1) of FIG. 3, since there is no table, the DTS that can be compared with the STC is the one associated with a frame to be decoded next. Thus, in the event of delayed termination of decoding of the first frame, the comparison between STC and DTS is delayed correspondingly with the result that the decoding of the second frame is delayed.
In the decoding operation (2) of FIG. 3, since there is a table, the STC can be compared with the DTSs in the table at regular intervals in synchronism with the decoding sync signal. Therefore, even if the termination of decoding of the first frame is delayed, the decoding of the first frame can be stopped at the time when decoding of the second frame is to begin, allowing the decoding of the second frame to be started immediately.
In the MPEG-based coding, use is made of three coding schemes: I-frame (corresponding to the I-picture) coding in which coding is performed on individual frames and a picture is reconstructed using only data from a single frame, P-frame (corresponding to the P-picture) coding in which use is made of predictive coding based on a past picture, and B-frame (corresponding to the B-picture) coding in which predictive coding is performed using both past and future frames. Since a P-frame is coded using prediction from one past I- or P-picture, the current P-frame has to be decoded using the past predicted picture. The decoding of a B-frame requires two past I- or P-pictures because two past and future I- or P-pictures are used in the prediction of it.
If the first frame is an I- or P-picture, stopping decoding of it will disable subsequent P- or B-frames from being displayed as pictures. Thus, if a picture is an I- or P-picture, the decoding of it is continued without being stopped as in the operation (1) shown in FIG. 3. When the second frame is a B-picture which is not used in decoding other pictures, the decoding of it is stopped so that decoding of the third frame is started.
Thus, by restricting pictures the decoding of which is to be stopped to B-frames, the effect of stopping the decoding process on display can be minimized. When DTSs are not appended to all frames, DTSs of frames with no DTS can be produced by adding the decoding time for 1 or 1.5 frames to the DTSs appended to frames. The resultant DTSs are entered into a management table. Only pictures with no DTSs can also be managed using a table. Special reproduction of pictures can also be realized through the use of a table.
The concurrent processing of multi-channel bit streams can be facilitated by creating a table for each channel. The provision of a table for each channel will accommodate a system having less decoders than there are channels to be processed. Thus, the provision of a table is very useful.
However, in the prior art a table is created for all the data that are written into the input buffer 16. The table thus must have the size corresponding to a maximum number of frames that can be written into the input buffer 16. As a result, the table size increases and the memory capacity required of the ID/DTS/storage location holding circuit 14 increases. The special reproduction of pictures, particularly reproduction in reverse direction, requires a very large amount of memory; that is, the input buffer is required to have a very large storage capacity.
According to an aspect of the present invention, there is provided an image processing device comprises: a reference time information extractor coupled to receive a bit stream containing coded image data and having system reference time information for reproducing the system reference time on an encoder side and decode start time information appended for extracting the system reference time information; a buffer coupled to receive the bit stream after the system reference time information has been extracted for storing it; a decoding controller coupled to the buffer for scanning a portion of the bit stream stored in the buffer to produce identification information for each frame of image data and then outputting the identification information together with decode start time information for the corresponding frame of image data, information concerning the storage location of the corresponding frame of image data in the buffer, and information concerning a coding type of the corresponding frame of image data; a hold circuit coupled to the decoding controller for holding the identification information, the decode start time information, the storage location information, and the coding type information for each frame of image data in the form of a mapping table; a reference time information revise circuit coupled to the reference time information extractor for producing reference time information on a decoder side and, in the even that there is a displacement between the decoder-side reference time information and the system reference time information, revising the decoder-side reference time information; a comparator coupled to the reference time information revise circuit and the hold circuit for making a comparison between the decoder-side reference time information and the decode start time information for each frame in the hold circuit and, when a match occurs, outputting the identification information for image data to be decoded; and a decoder coupled to the buffer and the decoding controller for reading that image data to be decoded from the buffer and decoding it, the decoding controller reading from the hold circuit the storage location information for the image data corresponding to the identification information output from the comparator and sending it to the decoder as an address used in reading the image data from the buffer.
According to another aspect of the present invention, there is provided an image processing method comprises: receiving a bit stream containing coded image data and having system reference time information for reproducing the system reference time on an encoder side and decode start time information appended and extracting the system reference time information from the bit stream; producing reference time information on a decoder side and, in the even that there is a displacement between the decoder-side reference time information and the system reference time information, revising the decoder-side reference time information; storing the bit stream after the system reference time information has been extracted into a buffer; scanning a portion of the bit stream stored in the buffer to produce identification information for each frame of image data and then holding, in the form of a mapping table, the identification information, decode start time information for the corresponding frame of image data, information concerning the storage location of the corresponding frame of image data in the buffer, and information concerning a coding type of the corresponding frame of image data; making a comparison between the decoder-side reference time information and the decode start time information for each frame in the table and, when a match occurs, outputting the identification information for image data to be decoded, and reading the storage location information for the image data corresponding to the identification information from the table and sending it to the decoder as an address used in reading and decoding the image data from the buffer.